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Biography 

Bob Lyons is an electronic commerce consultant with Unidex Inc., where he helps 
clients develop electronic commerce solutions, including extranet appUcations and 
EDI servers. Bob developed XML Convert, which is a free tool that converts flat 
file data to XML documents and vice versa. He has over 12 years of electronic 
commerce experience. Bob has led electronic commerce implementations at large 
corporations, given numerous presentations on electronic commerce, and has 
pubUshed articles on EDI, the Internet and X.400. 

Introduction 

Companies are beginning to use XML to send application data to browsers and to 
business applications. XML is well suited for the interchange of data, since XML 
documents are self-describing, easily parsed and can represent complex data 
structures. Also, there is a wide variety of high-quality, inexpensive tools for 
parsing and transforming XML documents. When using XML for data interchange, 
ideally, the sending appUcation will be able to export an XML document, and the 
receiving application will be able to import an XML document. Unfortunately, 
many legacy applications use flat files to import or export data. So, companies will 
need to convert flat files into XML documents when sending data to XML-capable 
applications. Likewise, companies will need to convert XML documents into flat 
files that can be imported into legacy applications. 

Flat Files 

Flat files contain machine-readable data that is typically encoded as printable 
characters. A flat file usually contains a series of records (or lines), where each 
record is a sequence of fields. A field contains an atomic piece of data (e.g., a 
postal code). 

Let's look at a simple flat file containing employee data. The file contains one or 
more employee records. Each record contains the following three fields: 



Employee's social security number (ssn) 

Employee's fiiU name (last name followed by a comma followed by 
a space followed by the first name) 
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Employee's salary 
The following are the contents of the employees flat file: 

123456789, "Carr, Lisa" , 100000 . 00 
444556666, "Barr, Clark" , 87000 . 00 
777227878, "Rabbitt, Jack" , 123000 . 00 

Each record contains information about one employee. The format of the flat file is 
Comma Separated Value (CSV), which means that each record is terminated by the 
operating system's line separator and the fields within a record are separated by a 
comma. In addition, a field value may be enclosed in quotes, which escape any 
Line Se arator terminator characters that appear within the field value. Note that 

- ^ the quotes that surround the field value are not actually part of the field value. 

Uuo es ^jg^^ ^ g^j^ value contains a quote character, then the field value must be 

surrounded by quotes and the quote character in the field value must escaped by 
prefixing it with an additional quote. 

Let's look at a more complicated flat file, the structure of which is similar to the 
structure of a Windows initialization file. The flat file contains a list of contacts. 
The following are the contents of the contacts flat file: 
[contact] 

name: Nancy Magi 11 

email : lil . magill@blackmountainhills . com 
phone: (100) 555-9328 
[contact] 

email : molly . j ones@oblada . com 
name: Molly Jones 

[contact] 

ptione: (200) 555-3249 
name: Penny Lcuie 

email : plane@bluesuburbans)cies . com 

Each contact consists of a begin contact record followed by three optional records. 
The begin contact record consists of the string "[contact]". The three optional 
records, which can appear in any order, are as follows: 



Comma Separated 
Value 
I Lin( 
I Quotes I 



^ Example Flat 
Filel^ 



The name record, which contains the full name of the contact. This 
record begins with a "name=" label followed by the name. 

The email record, which contains the email address of the contact 
This record begins with a "email=" label followed by the email 
address. 



The phone record, which contains the phone number of the contact. 
This record begins with a "phone=" label followed by the phone 
number. 

^ Line Separator » Each record in the flat file is a line that is terminated with the operating system's 
line separator. 

You might be wondering why these two files are considered "flat". The term "flat" 
Flat means that the file is not indexed. The term also implies that a flat file does not 

Hierarchical have a hierarchical structure; however, many flat files do have a hierarchical 
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^ Nested Groups 
of Records 1^ 



structure. Even simple flat files, such as the employees file above, contain a 
sequence of records where each record contains a sequence of fields. Many flat 
files, such as those used to exchange insurance claims, have more complicated data 
structures, such as multiple record types, groups of records, nested groups of 
records, repeating groups, etc. 

Flat files are commonly used to transfer data between two applications, since many 
business applications (e.g., SAP's R/3, EDI translators, legacy applications, etc.) 
use flat files to import and export data. For example, when a company receives an 
EDI invoice fi*om a vendor, it will use an EDI translator to convert the invoice data 
firom the EDI data format (e.g., XI 2) into the data format required by the accounts 
payable system. The EDI translator will produce a flat file containing the converted 
invoice data. The accoimts payable system then imports this flat file. 

In the future, many business appHcations will be able to import and export XML. 
For example, SAP has announced that R/3 will be able to import and export XML, 
in addition to importing and exporting the SAP IDOC flat files. Until then, there 
will be a need for conversion tools that can convert complex flat files into XML 
documents, and vice versa. 

Conversion Between Flat Fttes and XML 

Companies will need to convert flat files to XML when transferring data from 
^ Convert Flat File legacy applications to XML-capable appUcations (e.g., SAP's R/3, Microsoft's IE 
to XML » 5.0, etc.). Companies will also convert flat files to XML when they need to display 

the flat file data on a non-XML-capable browser, since it is easy to convert XML 

to HTML using XSLT. 



^ Export Flat File 
^ Import Flat File 



^ Conversion 
Tools P 
Export XML 
Import XML 



^ Convert XML to 
Flat File » 



^ Conversion 
Tools ^ 
Schema-Driven 



Companies will need to convert XML into flat files when transferring data firom 
XML-capable systems to a legacy system. 

Conversion between flat file data and XML can be done via generic conversion 
tools or custom scripts. Generic conversion tools are schema-driven, so fliat they 
can handle a wide range of flat file formats. Such a conversion tool uses the 
schema of the flat file in order to parse the file and convert it to an XML document. 
The conversion tool also needs the flat file schema when converting an XML 
document into a flat file that conforms to the flat file schema. 

The XFlat Language 

XFlat is an XML language for defining flat file schemas. An XFlat schema is an 
XML document that conforms to the XFlat language and that describes the format 
of a flat file. An XFlat schema defines the structure and syntax of a flat file that 
contains non-XML data. An XFlat schema also defines the structure and syntax of 
a nXFlat instance . An XFlat instance is an XML document whose structure is the 
same as the flat file and whose data is the same as the data in the flat file. In other 
words, an XFlat schema describes the structure of a flat file and the corresponding 
XFlat instance. 

Delimited xhe flat file that is described by an XFlat schema must consist of records, where 

^ Field m each record is a sequence of fields. A field is an atomic piece of data (e.g., a postal 

^ Field Separator » code). Records and fields may be delimited. A record separator (i.e., delimiter) 
Fixed length occxirs at the end of a record and helps the parser determine where the record ends. 
^ Record P Likewise, a field separator occurs at the end of a field and helps a parser to 



XFlat 

XFlat Instance 
XFlat Schema 
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^ Record 
Separator &• 

^ Nested Gix)ups 
of Records ^ 
Recursion 
Recursive Data 
Staictures 
Subgroups 



determine where a field ends. Fields that are not delimited must be fixed length 
(i.e., the minimum length of the field must be equal to flie maximum length of the 
field). 



The records may be grouped, and groups of records may be nested in a hierarchical 
structure (in other words, groups of records may contain subgroups). Note that 
XFlat does not support recursive data structures. 



XFlat Element 
Types 



The XFlat element types are as follows: 



XFlat Element 



XFlat, which is used to define an XFlat schema. The XFlat element 
is always the document element (i.e., root element) of an XFlat 
schema. An XFlat element must contain exactly one subelement, 
and this subelement must be a SequenceDef element, a ChoiceDef 
element or a RecordDef element. 



SequenceDef 
Element 



SequenceDef, which is used to define a sequence of objects, 
where each object may be a sequence, a choice or a record. A 
SequenceDef element must contain one or more subelements; 
each of these subelements must be a SequenceDef element, a 
ChoiceDef element or a RecordDef element. 



ChoiceDef, which is used to define a choice of one of a set of 
ChoiceDef objects, where each object may be a choice, a sequence or a 
Element record. A ChoiceDef element must contain one or more 

subelements; each of these subelements must be a SequenceDef 
element, a ChoiceDef element or a RecordDef element. 

RecordDef RecordDef, which is used to define a record, which is essentially a 
Element sequence of fields. A RecordDef element must contain one or 

more FieldDef elements. 



^ FieldDef FieldDef, which is used to define a field. A field contains an 
Element m atomic piece of data. A FieldDef element may not contain any 

subelements. 



^ MapToXml 
Attribute & 



Declarative 
Language 



An XFlat schema contains all the information needed to convert a flat file to XML 
(or vice versa). The MapToXml attribute in the XFlat language allows you to map 
each group, record and field to an XML element or to nothing. A field can also be 
mapped to an XML attribute. 

Note that XFlat is a declarative language. A non-programmer who is familiar with 
flat files can create an XFlat schema. 
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Download XML 

^^i^^^^J^ For more information about XFlat (e.g., definitions of the XML attributes 

^ Per More ^Flat language), please download XML Convert at 

Information m http://www.unidex.com/ and see the docxmientation that accompanies the 

^ http://www.unidexxom/ application. 



Example XFlat Schemas 

Let's look at the XFlat schema for the employees flat file. The contents of that file 
were as follows: 

123456789, "Carr, Lisa" , 100000 . 00 
444556666, "Barr, Clark" , 87000 . 00 
777227878, "Rabbitt, Jack" , 123000 . 00 

^ Example XFlat 

Schema » The following XFlat schema describes the layout of the employees flat file: 

<XFlat Ncune="employees_scheina" Descriptions "CSV flat file"> 

<SequenceDef Name=" employees" Descriptions "employees flat file"> 

<RecordDef Name=" employee" FieldSep="," RecSep="\\N" MaxOccur="0"> 
<FieldDef Name="ssn" NullAllowed="No" 

MinFieldLength="9" MaxFieldLength= " 9 " 
DataType=" Integer" MinValue="0" 
Quot edValue= " Yes " / > 
<FieldDef Name="name" NullAllowed="No" 

QuotedValue= "Yes " / > 
<FieldDef Name=:" salary" NullAllowed="No" 

DataType=" Float" MinValue="0" 
QuotedValue= "Yes " /> 

</RecordDef > 
< / S equenceDe f > 
</XFlat> 

Please note the following about this XFlat schema: 

Each of the following is mapped to an XML element in the XFlat 
instance by defauh: the SequenceDef element for the flat file, the 
Name Attribute RecordDef for the employee record and the FieldDef elements for 
the three fields. The tags for the XFlat instance are specified by the 
Name attributes in the SequenceDef, ChoiceDef, RecordDef and 
FieldDef elements. 

• 

MaxOccur The RecordDef element contains the MaxOccur="0" attribute, 
Attribute which means that there is no upper limit on the number of the 

employee records that may appear in the flat file. 

• 

Description The Description attribute, which is ixsed in the XFlat, 
Attribute SequenceDef and RecordDef elements, contains firee form text and 

is treated as a comment. 



Attribute The record separator for the record is defined as "\\N", which is 
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Normalization 
^ Line Separator 

RecSep Attribute 
^ Record 
Separator 
^XFlat 
Encoding 8^ 



an XFlat encoding for the line separator for the local operating 

system. If we were to define the value of the RecSep attribute as 
"&#D;&#A;", then an XML parser would convert this value to a 
space character during the attribute normalization process. XFlat , 
has encodings for a handful of special characters that are affected 
by attribute normalization. On Unix, "\\N" is converted to the line 
feed character (Unicode #xA). On Windows, "\\N" is converted to 
the carriage retum character (Unicode #xD) followed by the line 
feed character (Unicode #xA). 



^CSVS> 
^ FieldDef 
Element 9^ 
FieldSep 
Attribute 
QuotedValue 
Attribute 
^ Quotes 



All the FieldDef elements contain the QuotedValue="Yes" 
attribute, since the flat file format is CSV. For the same reason, 
the RecordDef element contains the FieldSep="," attribute. 



NuUAllowed All three FieldDef elements contain the NullAllowed="No" 
Attribute attribute, since all three fields are mandatory (i.e., the length of 

the field value must be greater than or equal to one character). 

Data Type The data type of the Social Security Number (ssn) field is declared 
as "Integer". 

Minimum Vahie The minimum value of the Social Security Number (ssn) field is 
set to zero, since a negative social security number is invalid. 

The data type of the salary field is declared as "Float". 

The minimum value of the Salary field is set to zero, since a 
negative salary is invalid. 

MaxFieklLength The FieldDef elements for the name and salary fields do not 
Atbibute include the MaxFieldLength attribute. Thus, the default 

maximxmi field length (i.e., 80 characters) applies to both fields. 

Now let's look at the XFlat schema for the contacts flat file. The following were the 
contents of that file: 



[contact] 

name: Nauicy Magi 11 

email : lil .magi ll@blackmountainhills .com 
phone: (100) 555-9328 
[contact] 

email : molly . j ones@oblacla . com 
name: Molly Jones 
[contact] 
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phone: (200) 555-3249 
name: Penny Lane 

email: plane@bluesiiburbanskies.com 

^ Example XFlat 

Schema » The following XFlat schema describes the layout of the contacts flat file: 

<XFlat Name="contacts_schema" Description= "unordered records "> 
<SequenceDef Name=" contacts "> 

<SequenceDef Name=" contact" MinOccur="0" MaxOccur="0"> 

<RecordDef Name="begin_contact" MapToXml="No" RecSep="\\N"> 
<FieldDef Name="begin_contact" 

ValidValue=" [contact] " MapToXml="No"/> 

</RecordDef > 

<ChoiceDef Name="unordered_recs" MapToXmls^No" 
MinOccur="0" MaxOccur="3"> 
<RecordDef Name="full_name" RecSep="\\N" MapToXml="No"> 
<FieldDef Name=" label" 

MinFieldLength= " 5 " MaxFieldLength= " 5 " 
ValidValue="name=" MapToXml="No"/> 
<FieldDef Name="full_name"/> 
</RecordDef > 

<RecordDef Name="phone_num" RecSep="\\N" MapToXml="No"> 
<FieldDef Name=" label" 

MinFieldLength= " 6 " MaxFieldLength= " 6 " 
ValidValue="phone=" MapToXml=*»No"/> 
<FieldDef Name="phone_nxjmber"/> 
</RecordDef > 

<RecordDef Name="email" RecSep="\\N" MapToXml="No"> 
<FieldDef Name=" label" 

MinFieldLength= " 6 " MaxFieldLength= " 6 " 
ValidValue=" emails" MapToXml="No"/> 
<FieldDef Name="email_address"/> 
</RecordDef > 
</ChoiceDef > 
< / S equenceDef > 
< / S equenceDef > 
</XFlat> 

Please note the following about this XFlat schema: 



^ Line 
Separator » 
41 Record 

Separator 
^ XFlat 
Encoding 



The record separator for the record is defined as "\\N", which is an 
XFlat encoding for the line separator for the local operating 
system. 



^MapToXml 
Attribute 



The FieldDef elements for all the label fields contain the 
MapToXml="No" attribute, since we don't want to map the label 
fields to XML. 



^ MapToXml AH four RecordDef elements contain the MapToXml="No" 
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Attribute e> element, since there is only one interesting field in each of these 
four record types. 



^ Field 
Separator & 
Fixed Length 
Field 
^ Record 
Separator P> 
Variable Length 
Field 



Each of the RecordDef elements within the ChoiceDef element 
consists of a fixed length field (with no field separator) followed 
by a variable length field that is delimited by the record separator. 



<i Convert Flat File 

to XML S> 

^ Convert XML to 

Flat File » 

^ Java Application 
P 

Next Version 



This schema does not describe all the syntax rules for this flat file. 
For example, a contact is not allowed to contain two name records; 
unfortunately, the XFlat language is not yet capable of expressing 
such a syntactical constraint. 

XML Convert 

XML Convert is a free Java appUcation that uses XFlat schemas to convert flat 
files into XML, and vice versa. XML Convert 1.1, which is the version currently 
available at http://www.unidex.com/, converts flat files into XML, but does not 
convert XML into flat files. The next version, which converts in both directions 
and which is described in this paper, will be available shortly at 
http://www.imidex.com/. 



Key Features j^^^ features of XML Convert include: 



Fixed Length 

Human-Readable Handles a wide range of flat file formats (e.g., variable length 
Report records, CSV, fixed length records, files that contain multiple 

Semi-Structured record types, nested groups of records, records that contain a 
Data mixed of delimited and non-delimited fields, records in which 

Variable Length each field has a different delimiter, etc.). XML Convert can even 
Wide Range of handle semi-structured data, such as a human-readable report that 
Flat File Foraiats contains rows and colunms of data. 



XML Language ^^^^ ^ ^.^p^^ language (i.e., XFlat) for the flat file schemas. 
Portability ^^^^^ ^ j^^^ portability. 
^ Java 
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Application i> 

Windows 
Executable 



Includes a Windows executable for ease of use on a PC. XML 
Convert can also be invoked as a Java application from the 
command line. 



4 API, 
Application 
Programming 
Interface 1^ 



Includes an API, so that XML Convert can be easily invoked 
from the user's Java application. 



^ , , Includes the com.xmidex.xflat.XflatOutputHandler class 

OutputDocumentHandler ^^^^^^^^ the 

com.jclark.xsl.sax.OutputDocumentHandler interface. 
When the com.unidex.xflat.XflatOutputHandler class is 
specified as the output method in an XSLT stylesheet, XT 
will pass the result tree to XML Convert, which will 
validate the result tree against the XFlat schema and, if the 
result tree conforms to the XFlat schema, produce a flat 
file. 



Interface 

^ XSLT Stylesheet 
^XT& 

^ XflatOutputHandler 
Class e> 



<i Validation of 
DataS> 



Validates the flat file or XFlat instance against the XFlat schema. 
If an error is found in the input file (i.e., the flat file or the XFlat 
instance), then the conversion process is terminated and a detailed 
error message is generated. 



Error Messages 



Error messages include a detailed description of the error and the 
location of the error (within the XFlat schema file, the XFlat 
instance or the flat file), so that the user can quicldy troubleshoot 
the error. 



Large Flat Files 
Stream Mode 



Converts in stream (i.e., serial) mode, which means that XML 
Convert does not read the entire flat file or XFlat instance into 
memory. Thus, XML Convert can convert very large flat files into 
XML, and vice versa. 



XML Convert is free. 

^ Convert Flat File when XML Convert transforms a flat file to an XML document (i.e., an XFlat 
to XML 
^ Validation of 
DataS^ 



^ Convert XML to 
Flat File ^ 



instance), it will verify the structure of the flat file data and the data types of the 
fields using the XFlat schema. If the flat file does not pass this verification, then it 
is rejected. This verification minimizes the chance that an invalid XML document 
will be sent to the receiving application. 

Likewise, when XML Convert transforms an XML document to a flat file, it will 
verify that the resulting flat file conforms with the XFlat schema. This verification 
minimizes the chance that an invalid flat file will be imported into a business 
appUcation. 
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Converting Between the Employees Flat File and XML 

^ Convert Flat File Using the XFlat schema for the employees flat file (see above), XML Convert 
to XML » would convert the employees flat file into the following XML docximent (i.e., 

XFlat instance): 

<?xml versions •l,0'?> 
<en!ployees> 

<einployee> 

<ssn>123456789</ssn> 

<name>Carr , Lisa< /name> 

<salary>100000 . 00</salary> 
</employee> 
<employee> 

<ssn>444556666</ssn> 

<name>Barr , Clark</naine> 

<salary>87000 . 00</salary> 
</einployee> 
<einployee> 

<ssn>777227878</ssn> 

<name>Rabbitt , Jack</naine> 

<sala2:Y>123000 . 00</salary> 
</employee> 
</ employees > 

FW^I^D.^^^^ reverse direction, using the same XFlat schema, XML Convert would 

convert this XFlat instance back into the original employees flat file. 

Converting Between the Contacts Flat File and XML 

7 YMT e> ^^^S the XFlat schema for the contacts flat file (see above), XML Convert would 

to XML » convert the contacts flat file into the following XML document: 

<?xml versions* 1.0 •?> 
<contacts> 

<contact> 

<full_naine>Nancy Magill</full_name> 

<eniail_address>lil .magill@blackmountainhills . com</einail_address> 

<phone_niiinber> (100) 555-9328</phone_nuniber> 
</ contact > 
<contact> 

<einail_address>molly . j ones®oblada . com</ einail_address> 

<full_name>Molly Jones </full_name> 
</contact> 
<contact> 

<phone_number > (200) 555-3249< /phone^nuitODer > 
<full_naine> Penny Lane</f ull_naine> 

<email_address>plane@bluesuburbanskies . coin</einail_address> 
</ contact > 
</contacts> 

Fn^^T^ D^^ reverse direction, using the same XFlat schema, XML Convert would 

Hat t lie B> convert this XFlat instance back into the original contacts flat file. 

Using XSLT With XML Convert 

After converting a flat file into XML using XML Convert, it may be necessary to 
change the structure and/or element names of the resulting XML document (i.e., 
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the XFlat instance) before sending it to the receiving application. For example, if 
^ Convert XML to the resulting XFlat instance will be sent to a browser that does not support XML, 
Flat File 8* then the XFlat instance should be converted from XML to HTML using an XSLT 

^ XSLT processor. If the output will be sent to an XML-capable application, then it will 

^ XSLT Processor Probably be necessary to use an XSLT processor to convert the XFlat instance into 

a new XML document whose structure meets the requirements of the receiving 

application. (Note that the output of the XSLT processor can be an XML document 

or an HTML document.) 

If the resulting XFlat instance will be sent to an XML-capable browser, then the 
XFlat instance can specify a stylesheet, so that the browser renders the XML 
document as a nicely formatted web page. 

When converting an XML document into a flat file, the XML document would 
probably not have the same structure as the target flat file. In this case, the user can 
^ Convert XML to use an XSLT processor to convert the XML document into an XFlat instance (i.e.. 
Flat File e» an XML document whose structure is the same as the structure of the target flat 

file). The user would then employ XML Convert to transform the XFlat instance 
into a flat file. XML Convert uses an XFlat schema to parse the XFlat instance and 
produce the target flat file. 

vf tl^ If you are using XT to convert the XML document into an XFlat instance, then 

^ XT% the XSLT stylesheet can specify the com.unidex.xflat.XflatOutputHandler class 

-ssYfi n f, +H 11 as the output method, so that XT passes the result tree to XML Convert, which 
^ AiiatUutputHanaiei ^.^ ^.^g^^ j^e XFlat schema and, if the result tree 

^^^^ ^ conforms to the XFlat schema, produce a flat file. 

Q 11 following is an example of an XSLT stylesheet that specifies the 

Stylesheet » com.unidex.xflat.XflatOutputHandler class as the output method: 

<xsl : stylesheet 

xmlns : xsl= "http : //www . w3 . org/XSL/Transf orm/l . 0 " > 

<xsl:output method="xtj : com. unidex. xflat .XflatOutput Handler" 

xmlns : xt j = "http : //www . j dark . com/xt/ j ava" /> 
<l-- The rest of the stylesheet goes here. 
</xsl : stylesheet> 

Transtbrmation Note that XML Convert and the XFlat language do not provide any XML to XML 
Features transformation features, since XSLT can be used to do XML to XML 

transformation. 

Also note that an XSLT processor can convert an XML document into non-XML 
text, without any help fi-om XML Convert. Thus, you could use an XSLT processor 
without XML Convert to transform an XML document into a flat file. However, it 
^ CSV ^ would be very difficult to write an XSLT stylesheet that syntactically vaUdates the 

Field Lengtli Fields resulting flat file. If s important to validate the resulting flat file, so that the 
^ XSLT Processor receiving appHcation does not import an invalid flat file. Also, it would be difficult 
B> to write a stylesheet that produces a CSV flat file, since the stylesheet would have 

to escape any quote characters that are embedded in the field values. It would also 
be difficult to write a stylesheet that produces a flat file containing fixed length 
fields, since the stylesheet would have to pad the values of some fields with spaces, 
so that the length of each field is correct. 

Summary 
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XML Convert is a free Java application that uses XFlat schemas to convert flat files 
into XML and vice versa. XFlat is an XML language for defining flat file schemas. 
XML Convert uses an XFlat schema to parse and validate the input file (i.e., the 
flat file or the XFlat instance), and to produce the output file. XML Convert 
supports a wide variety of flat file formats, including CSV, semi-structured data 
(e.g., human readable reports), fixed length records and fields, multiple record 
types, groups of records, nested groups, etc. 

<i For More 

Infonnation » ^ot more information about XML Convert and XFlat, please see 

^ http://vvww.unidex.com/ http://www.unidex.com/. 
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